ABSTRACT
Introduction: During the recent covid-19 vaccination campaign, the number of ICSRs reported by patients and professionals has dramatically increased, reaching up to almost 1 M declarations only in Europe (EMA numbers). To deal with such growing amount of data, Synapse Medicine-, in collaboration with The French National Agency for Medicines and Health Products Safety (ANSM), have developed an artificial intelligence (AI) tool, the Medication Shield, which, based on a natural language processing algorithm, is able to detect ADRs from patients' reports and to code them into an appropriate MedDRA preferred term (PT). Before the covid-19 pandemic, this system was successful in detecting ADRs from the patient reports declared through the French web national reporting system (1, 2). However, how it behaves in conditions of higher reporting flow rate is unknown at present. Objective(s): To evaluate the performance of the Medication Shield in detecting vaccine-related ADRs from patients' ICSRs declared across the covid-19 vaccination campaign. Method(s): A machine learning (ML) pipeline composed by a light Gradient Boosting Machine ensemble model was employed to detect and code covid-19 vaccine-related ADRs from patients' ICSRs declared through the web reporting system during the vaccination campaign (Jan 2021-Apr 2022). The encoding of regional pharmacovigilance centers was employed as the reference ground truth to train the algorithm in a supervised manner. Moreover, a panel of three pharmacologists, with significant experience in ADRs encoding, was set-up to perform a case-by-case analysis of 200 hundreds reports for which the algorithm provided improper encoding. Result(s): Overall, 65.191 ICSRs were extracted and used to train our ML algorithm. Of this, 54.987 were employed to validate the system. Importantly, almost 86% of the ICSRs were related to covid vaccines. Because the percentage of newly reported ADRs increased over time and was higher for vaccine than not-vaccine related reports, we split the training and validation sets in batches with similar ADRs distribution. Performance evaluation is currently under process. Initial feedbacks from the analysis performed by the experts are showing an uneven distribution of false positive and false negative across samples. Results from the other experts are needed to confirm this finding. Conclusion(s): The core findings of this study will be gathered in the forthcoming weeks and be ready for the ISoP meeting in September. This work will provide new insights about the effectiveness of deploying AI as a support to treat real world data in a context of sanitary crisis.
ABSTRACT
Introduction: During the recent covid-19 vaccination campaign, the number of ICSRs reported by patients and professionals has dramatically increased, reaching up to almost 1 M declarations only in Europe (EMA numbers). To deal with such growing amount of data, Synapse Medicine®, in collaboration with The French National Agency for Medicines and Health Products Safety (ANSM), have developed an artificial intelligence (AI) tool, the Medication Shield, which, based on a natural language processing algorithm, is able to detect ADRs from patients' reports and to code them into an appropriate MedDRA preferred term (PT). Before the covid-19 pandemic, this system was successful in detecting ADRs from the patient reports declared through the French web national reporting system (1, 2). However, how it behaves in conditions of higher reporting flow rate is unknown at present. Objective: To evaluate the performance of the Medication Shield in detecting vaccine-related ADRs from patients' ICSRs declared across the covid-19 vaccination campaign. Methods: A machine learning (ML) pipeline composed by a light Gradient Boosting Machine ensemble model was employed to detect and code covid-19 vaccine-related ADRs from patients' ICSRs declared through the web reporting system during the vaccination campaign (Jan 2021-Apr 2022). The encoding of regional pharmacovigilance centers was employed as the reference ground truth to train the algorithm in a supervised manner. Moreover, a panel of three pharmacologists, with significant experience in ADRs encoding, was set-up to perform a case-by-case analysis of 200 hundreds reports for which the algorithm provided improper encoding. Results: Overall, 65.191 ICSRs were extracted and used to train our ML algorithm. Of this, 54.987 were employed to validate the system. Importantly, almost 86% of the ICSRs were related to covid vaccines. Because the percentage of newly reported ADRs increased over time and was higher for vaccine than not-vaccine related reports, we split the training and validation sets in batches with similar ADRs distribution. Performance evaluation is currently under process. Initial feedbacks from the analysis performed by the experts are showing an uneven distribution of false positive and false negative across samples. Results from the other experts are needed to confirm this finding. Conclusion: The core findings of this study will be gathered in the forthcoming weeks and be ready for the ISoP meeting in September. This work will provide new insights about the effectiveness of deploying AI as a support to treat real world data in a context of sanitary crisis.